Modeling of Web Robot Navigational Patterns

In recent years, it is becoming increasingly diÆcult to ignore the impact of Web robots on both commercial and institutional Web sites. Not only do Web robots consume valuable bandwidth and Web server resources, they are also making it more diÆcult to apply Web Mining techniques e ectively on the Web logs. E-commerce Web sites are also concern about unauthorized deployment of shopbots for the purpose of gathering business intelligence at their Web sites. Ethical robots can be easily detected because they tend to follow most of the guidelines proposed for robot designers. On the other hand, unethical robots are more diÆcult to identify since they may camou age their entries in the Web server logs. In this paper, we examine the problem of identifying Web robot sessions using standard classi cation techniques. Due to the temporal nature of the data, the classi cation model may vary depending on the number of requests made by the Web user or robot. Our goal is to determine the minimum number of requests needed to distinguish between robot and non-robot sessions, with reasonably high accuracy. Our preliminary results show that highly accurate models can be obtained after three requests using a small set of access features computed from the Web server logs.